1 Likelihood-Based Inference

Setting: ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }{p}_{\theta }(x)}$, ${\displaystyle p_{\theta }(x)}$ is "smooth" in ${\displaystyle \theta }$. Assume ${\displaystyle {\mathbb{E}}_{\theta }\mathrm{\nabla }{l}_1(\theta ;X_i)=0}$, ${\displaystyle {\mathrm{Var}}_{\theta }[\mathrm{\nabla }{l}_1(\theta ;X_i)]=-{\mathbb{E}}_{\theta }{\mathrm{\nabla }}^2{l}_1(\theta ;X_i)=J_1(\theta )}$ positive definitive. (see Fisher information), and MLE is consistent: ${\displaystyle {\hat{\theta }}_{\mathrm{MLE}}\stackrel{p_{\theta }}{\to }\theta }$. Then if ${\displaystyle \theta ={\theta }_0}$, $$\begin{array}{rl}\frac{1}{\sqrt{n}}\mathrm{\nabla }{l}_n({\theta }_0;X)& \Rightarrow {\mathcal{N}}_d(0,J_1({\theta }_0)),\\ \frac{1}{n}{\mathrm{\nabla }}^2{J}_n({\theta }_0;X)& \stackrel{p}{\to }{J}_1({\theta }_0).\end{array}$$
Since we used $$0=\mathrm{\nabla }{l}_n({\hat{\theta }}_n)\approx \mathrm{\nabla }{l}_n({\theta }_0)+{\mathrm{\nabla }}^2{l}_n({\theta }_0)({\hat{\theta }}_n-{\theta }_0)$$ to get ${\displaystyle \sqrt{n}({\hat{\theta }}_n-{\theta }_0)\Rightarrow {\mathcal{N}}_d(0,J_1({\theta }_0{)}^{-1})}$, then we can use this for inference on ${\displaystyle {\theta }_0}$!

1.1 Wald-Type Confidence Regions

Assume we have some estimator ${\displaystyle {\hat{J}}_n\succ 0}$ s.t. ${\displaystyle \frac{1}{n}{\hat{J}}_n\stackrel{p}{\to }{J}_1({\theta }_0)\succ 0}$, then we plug in: if ${\displaystyle \sqrt{n}({\hat{\theta }}_n-{\theta }_0)\Rightarrow {\mathcal{N}}_d(0,J_1({\theta }_0{)}^{-1})}$, then $$(J_1({\theta }_0){)}^{\frac{1}{2}}\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\Rightarrow {\mathcal{N}}_d(0,I_d),$$ so by Slutsky's theorem $${\hat{J}}_n^{\frac{1}{2}}({\hat{\theta }}_n-{\theta }_0)\Rightarrow {\mathcal{N}}_d(0,I_d).$$ This leads to test of ${\displaystyle H_0:\theta ={\theta }_0}$: $$||{\hat{J}}_n^{\frac{1}{2}}({\hat{\theta }}_n-{\theta }_0)|{|}^2\Rightarrow {\chi }_d^2,$$ so ${\displaystyle {\mathbb{P}}_{{\theta }_0}({\hat{J}}_n^{\frac{1}{2}}({\hat{\theta }}_n-{\theta }_0)\ge {\chi }_d^2(\alpha ))\to \alpha }$.
So we reject ${\displaystyle {\theta }_0}$ iff ${\displaystyle ||{\hat{J}}_n^{\frac{1}{2}}(\widehat{{\theta }_n}-{\theta }_0)|{|}^2>{\chi }_d^2(\alpha )}$. ${\displaystyle ⟺}$ reject ${\displaystyle {\theta }_0}$ iff ${\displaystyle {\theta }_0\notin \underset{\text{confidence ellipsoid}}{\underbrace{{\hat{\theta }}_n+{\hat{J}}_n^{-\frac{1}{2}}{B}_{{\chi }_d^2(\alpha )}(0)}}}$. ${\displaystyle {\hat{J}}_n^{-\frac{1}{2}}=O_p(1/\sqrt{n})\approx n^{-\frac{1}{2}}{J}_1({\theta }_0{)}^{-\frac{1}{2}}}$.
For ${\displaystyle d=1}$, we reject ${\displaystyle {\theta }_0}$ iff ${\displaystyle |\sqrt{{\hat{J}}_n}({\hat{\theta }}_n-{\theta }_0)|>z_{\frac{\alpha }{2}}}$. ${\displaystyle ⟺}$ reject ${\displaystyle {\theta }_0}$ iff ${\displaystyle {\theta }_0\notin {\hat{\theta }}_n\pm {\hat{J}}_n^{-\frac{1}{2}}{z}_{\frac{\alpha }{2}}}$.

More info ${\displaystyle ⟺}$ smaller ellipse (shrinks like ${\displaystyle 1/\sqrt{n}}$)

Options for ${\displaystyle {\hat{J}}_n}$:

1. Most obvious to plug in MLE for ${\displaystyle J_n(\theta )}$: $$\begin{array}{rl}{\hat{J}}_n& =J_n({\hat{\theta }}_n)={\mathrm{Var}}_{\theta }(\mathrm{\nabla }{l}_n(\theta ;X)){|}_{\theta ={\hat{\theta }}_n}\ne {\mathrm{Var}}_{{\hat{\theta }}_n}(\mathrm{\nabla }{l}_n({\hat{\theta }}_n;X)).\\ \text{or }{\hat{J}}_n& =-{\mathbb{E}}_{\theta }{\mathrm{\nabla }}^2{l}_n(\theta ){|}_{\theta ={\hat{\theta }}_n}.\end{array}$$
2. Observed Fisher information: ${\displaystyle {\hat{J}}_n=-{\mathrm{\nabla }}^2{l}_n({\hat{\theta }}_n;X)}$.

Both have ${\displaystyle \frac{1}{n}\stackrel{p}{\to }\frac{1}{n}{J}_n({\theta }_0)}$ (under regularity; cts second derivative; MLE consistent).
Both make sense outside of iid setting: ${\displaystyle \frac{{\hat{J}}_n}{J_n}\stackrel{p}{\to }1}$.
Heuristically, plug-in measures info about ${\displaystyle \theta }$ in "typical" data set but observe info measures info about ${\displaystyle \theta }$ in "this" data set.

Wald interval for ${\displaystyle {\theta }_j}$: $$\begin{array}{rl}& {\hat{\theta }}_n\approx N_d({\theta }_0,J_n({\theta }_0{)}^{-1}),\\ & {\hat{\theta }}_{n,j}\approx N_d\underset{\mathrm{se}({\hat{\theta }}_{n,j}{)}^2}{\underbrace{({\theta }_{0,j},(J_n({\theta }_0{)}^{-1}{)}_{jj})}},\\ & c_j={\hat{\theta }}_{n,j}\pm \sqrt{({\hat{J}}_n^{-1}{)}_{jj}}\cdot z_{\frac{\alpha }{2}}.\end{array}$$
Confidence ellipsoid: ${\displaystyle {\theta }_{0,S}=({\theta }_{0,j}{)}_{j\in S}}$, ${\displaystyle |S|=k\le d}$. ${\displaystyle {\hat{\theta }}_{n,s}\approx N_k({\theta }_{0,S},(J_n({\theta }_0{)}^{-1}{)}_{SS})}$, ${\displaystyle \to c_S={\hat{\theta }}_{n,S}+(({\hat{J}}_n^{-1}{)}_{SS}{)}^{\frac{1}{2}}{B}_{{\chi }_k(\alpha )}(0)}$.
More generally, if ${\displaystyle {\hat{\theta }}_n}$ is any consistent estimator with $$\sqrt{n}({\hat{\theta }}_n-{\theta }_0)\Rightarrow N_d(0,\mathrm{\Sigma }({\theta }_0)),$$ and we have $$\frac{1}{n}{\hat{\mathrm{\Sigma }}}_n\stackrel{p_{{\theta }_0}}{\to }\mathrm{\Sigma }({\theta }_0)>0,{\hat{\mathrm{\Sigma }}}_n^{-\frac{1}{2}}({\hat{\theta }}_n-{\theta }_0)\Rightarrow N_d(0,I_d).$$ (${\displaystyle {\hat{\theta }}_n}$ not necessarily MLE)

Taylor expansion of ${\displaystyle l_n}$: $$\begin{array}{rl}& {\hat{J}}_n^{\frac{1}{2}}({\hat{\beta }}_n-{\beta }_0)\Rightarrow N_d(0,I_d),\\ & {\hat{J}}_n=-{\mathrm{\nabla }}^{2}l({\hat{\beta }}_n;X).\end{array}$$

Advantages and Disadvantages
Advantages:

1. Easy to invert, simple confidence regions.
2. Asymptotically correct.
   Disadvantages:
3. Have to compute MLE.
4. Depends on parameterization.
5. Relies on two approximations: $$\mathrm{\nabla }{l}_n\approx \mathrm{Normal},l_n\approx \text{quadratic}.$$
6. Needs MLE to be consistent
7. Confidence interval/ellipsoid could go outside ${\displaystyle \mathrm{\Theta }}$.

1.2 Score Test

Test : ${\displaystyle H_0:\theta ={\theta }_0\text{ vs }{H}_1:\theta \ne {\theta }_0}$.
We can bypass quadratic approximation entirely by using score as test statistic

We could reject ${\displaystyle H_0:\theta ={\theta }_0}$ if $$||J_n({\theta }_0{)}^{-\frac{1}{2}}\mathrm{\nabla }{l}_n({\theta }_0;X)|{|}^2\ge {\chi }_d^2(\alpha )\stackrel{d=1}{⟹}\frac{{\dot{l}}_n({\theta }_0)}{\sqrt{J_n({\theta }_0)}}\sim N(0,1).$$ Can do 1-sided tests.

• No approx to ${\displaystyle l_n}$, no MLE.
• Don't need to estimate Fisher information at ${\displaystyle {\theta }_0}$.

Can be generalized to case with nuisance parameters. Typically estimate via MLE on ${\displaystyle {\mathrm{\Theta }}_0}$.

Score test is invariant to reparameterization: assume ${\displaystyle d=1}$, ${\displaystyle \theta =g(\xi )}$, ${\displaystyle g^{\prime }(\xi )>0,\mathrm{\forall }\xi }$. ${\displaystyle q_{\xi }(x)=p_{g(\xi )}(x)}$. $$\begin{array}{rl}{l}^{\prime (\xi )}(\xi ;x)& =\frac{\mathrm{d}}{\mathrm{d}\xi }\log p_{g(\xi )}(x)=l^{\prime (\theta )}(g(\xi );X)\cdot g^{\prime }(\xi ),\\ J^{(\xi )}(\xi )& =J^{(\theta )}(g(\xi ))\cdot g^{\prime }(\xi {)}^2,\end{array}$$ so $$\frac{l^{\prime (\xi )}({\xi }_0;X)}{\sqrt{J^{(\xi )}({\xi }_0)}}\stackrel{\mathrm{a}.\mathrm{s}.}{=}\frac{l^{\prime (\theta )}({\theta }_0;X)}{\sqrt{J^{(\theta )}({\theta }_0)}}$$ if ${\displaystyle {\theta }_0=g({\xi }_0)}$.

Note ${\displaystyle \sum _{i=1}^n{\pi }_i=1}$, so this is a full-rank ${\displaystyle (d-1)-}$ parameter exponential family. E.g. $${\pi }_j=\{\begin{array}{rl}& \frac{1}{1+\sum _{k>1}{\mathrm{e}}^{{\eta }_k}},j=1,\\ & \frac{{\mathrm{e}}^{{\eta }_j}}{1+\sum _{k>1}{\mathrm{e}}^{{\eta }_k}},j>1.\end{array}$$
So $$\begin{array}{rl}\mathrm{\nabla }{l}_n& =(N_2,\cdots ,N_d)-(n{\pi }_2,\cdots ,n{\pi }_d),\\ {\mathrm{Var}}_{\eta }(\mathrm{\nabla }l(\eta ))& =\left(\begin{array}{ccc}n{\pi }_2(1-{\pi }_2)& \cdots & -n{\pi }_i{\pi }_j\\ ⋮& \ddots & ⋮\\ -n{\pi }_i{\pi }_j& \cdots & n{\pi }_d(1-{\pi }_d)\end{array}\right)\\ & =n(\mathrm{diag}({\pi }_{2:d})-{\pi }_{2:d}{\pi }_{2:d}^{\mathrm{T}}),\\ \Rightarrow J_n(\eta {)}^{-1}& =\frac{1}{n}(\mathrm{diag}({\pi }_{2:d}{)}^{-1}-{\pi }_1^{-1}{11}^{\mathrm{T}}).\end{array}$$
Here we use $$(A+u{v}^{\mathrm{T}}{)}^{-1}=A^{-1}-\frac{A^{-1}u{v}^{\mathrm{T}}{A}^{-1}}{1+v^{\mathrm{T}}{A}^{-1}u}.$$ So the score test of ${\displaystyle H_0:\pi ={\pi }_0}$ is $$\mathrm{\nabla }{l}_n({\eta }_0)J_n^{-1}({\eta }_0)\mathrm{\nabla }{l}_n({\eta }_0)=\sum _{j=1}^d\frac{(N_j-n{\pi }_{0}j{)}^2}{n{\pi }_{0}j}\stackrel{P_{{\pi }_0}}{\Rightarrow }{\chi }_{d-1}^2.$$

2 Generalized LRT

Test ${\displaystyle H_0:\theta ={\theta }_0\text{ vs }{H}_1:\theta \ne {\theta }_0}$. Taylor expand around ${\displaystyle {\hat{\theta }}_n}$:

Test statistic: ${\displaystyle 2(l_n({\hat{\theta }}_n;X)-l_n({\theta }_0;X))\stackrel{P_{{\theta }_0}}{\Rightarrow }{\chi }_d^2}$.

Consider ${\displaystyle H_0:\theta \in {\mathrm{\Theta }}_0\text{ vs }{H}_1:\theta \in \mathrm{\Theta }\mathrm{\setminus }{\mathrm{\Theta }}_0}$, assume

• ${\displaystyle \mathrm{\Theta }={\mathbb{R}}^d}$, ${\displaystyle {\mathrm{\Theta }}_0}$ is a ${\displaystyle d_0}$ -dim manifold.
• ${\displaystyle {\theta }_0\in \text{relint}({\mathrm{\Theta }}_0)}$.
• ${\displaystyle {\hat{\theta }}_n\stackrel{P_{{\theta }_0}}{\to }{\theta }_0}$.
• Likelihood is "smooth".

Then ${\displaystyle 2(l_n({\hat{\theta }}_n)-l_n({\hat{\theta }}_0))\Rightarrow {\chi }_{d-d_0}^2}$, where ${\displaystyle {\hat{\theta }}_0=\arg \underset{\theta \in {\mathrm{\Theta }}_0}{min}{l}_n(\theta ;X)}$.

Why? Assume WLOG ${\displaystyle {\theta }_0=0}$, ${\displaystyle J_1(0)=I_d}$. Then ${\displaystyle {\hat{\theta }}_n\approx {\mathcal{N}}_d({\theta }_0,\frac{1}{n}{I}_d)}$. And locally, ${\displaystyle {\mathrm{\nabla }}^2{l}_n(\theta )\approx -n{I}_d}$ near ${\displaystyle {\theta }_0}$, $$\begin{array}{rl}{l}_n(\theta )-l_n({\hat{\theta }}_n)& \approx \frac{n}{2}||\theta -{\hat{\theta }}_n|{|}^2\\ {\hat{\theta }}_0& \approx \arg \underset{\theta \in {\mathrm{\Theta }}_0}{min}||\theta -{\hat{\theta }}_n||={\mathrm{Proj}}_{{\mathrm{\Theta }}_0}({\hat{\theta }}_n)\\ 2(l_n({\hat{\theta }}_0)-l_n({\hat{\theta }}_n))& \approx n||{\hat{\theta }}_n-{\mathrm{Proj}}_{{\mathrm{\Theta }}_0}({\hat{\theta }}_n)|{|}^2\\ & =n||{\mathrm{Proj}}_{{\mathrm{\Theta }}_0}^{\perp }(\widehat{{\theta }_{\mathrm{n}}})|{|}^2\Rightarrow {\chi }_{d-d_0}^2.\end{array}$$

3 Asymptotic Equivalence

Recall quadratic approximation picture (${\displaystyle d=1}$):

Then Wald: ${\displaystyle \approx ||{\hat{J}}_n^{\frac{1}{2}}({\hat{\theta }}_n-{\theta }_0)|{|}^2}$; Score: ${\displaystyle ||J_n({\theta }_0{)}^{-\frac{1}{2}}\mathrm{\nabla }{l}_n({\theta }_0)|{|}^2}$.

4 Asymptotic Relative Efficiency (ARE)

Suppose ${\displaystyle {\hat{\theta }}_n^{(i)},i=1,2}$ are two asymptotic normal estimators of ${\displaystyle {\theta }_0\in \mathbb{R}}$, with $$\sqrt{n}({\hat{\theta }}_n^{(i)}-{\theta }_0)\Rightarrow \mathcal{N}(0,{\sigma }_i^2).$$ The ARE of ${\displaystyle {\hat{\theta }}^{(2)}}$ w.r.t ${\displaystyle {\hat{\theta }}^{(1)}}$ is ${\displaystyle \frac{{\sigma }_1^2}{{\sigma }_2^2}}$. E.g. if ${\displaystyle {\sigma }_2^2=2{\sigma }_1^2}$ then ${\displaystyle {\hat{\theta }}^{(2)}}$ is 50% as efficient.
Interpretation: suppose ${\displaystyle \frac{{\sigma }_1^2}{{\sigma }_2^2}=\gamma \in (0,1)}$. Then for large ${\displaystyle n}$, $${\hat{\theta }}_{[{\gamma }_n]}^{(1)}(X_1,\cdots ,X_{[{\gamma }_n]})\stackrel{D}{\approx }{\hat{\theta }}_n^{(2)}(X_1,\cdots ,X_n)\approx \mathcal{N}(\theta ,\frac{{\sigma }_2^2}{n}).$$ Using ${\displaystyle {\hat{\theta }}^{(2)}}$ is like throwing away ${\displaystyle 100(1-\gamma )\mathrm{\%}}$ of the data and then using ${\displaystyle {\hat{\theta }}^{(1)}}$.